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Abstract 

Byzantine agreement algorithms typically assume implicit initial state consistency and synchroniza- 
tion among the correct nodes and then operate in coordinated rounds of information exchange to reach 
agreement based on the input values. The implicit initial assumptions enable correct nodes to infer 
about the progression of the algorithm at other nodes from their local state. This paper considers a 
more severe fault model than permanent Byzantine failures, one in which the system can in addition 
be subject to severe transient failures that can temporarily throw the system out of its assumption 
boundaries. When the system eventually returns to behave according to the presumed assumptions it 
may be in an arbitrary state in which any synchronization among the nodes might be lost, and each 
node may be at an arbitrary state. We present a self-stabilizing Byzantine agreement algorithm that 
reaches agreement among the correct nodes in an optimal ration of faulty to correct, by using only the 
assumption of eventually bounded message transmission delay. In the process of solving the problem, 
two additional important and challenging building blocks were developed: a unique self-stabilizing pro- 
tocol for assigning consistent relative times to protocol initialization and a Reliable Broadcast primitive 
that progresses at the speed of actual message delivery time. 

Categories and Subject Descriptors: C.2.4 [Distributed Systems]: Distributed applications; 
General Terms: Algorithms, Reliability, Theory. 

Keywords: Byzantine Agreement, Self-Stabilization, Byzantine Faults, Pulse Synchronization, Transient 
Failures, Reliable Broadcast. 



1 Introduction 

The Byzantine agreement (Byzantine Generals) problem was first introduced by Pease, Shostak and Lam- 
port [13]. It is now considered as a fundamental problem in fault-tolerant distributed computing. The task 
is to reach agreement in a network of n nodes in which up-to / nodes may be faulty. A distinguished node 
(the General or the initiator) broadcasts a value m, following which all nodes exchange messages until the 
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non-faulty nodes agree upon the same value. If the initiator is non-faulty then all non-faulty nodes are 
required to agree on the same value that the initiator sent. 

Standard deterministic Byzantine agreement algorithms operate in the synchronous network model 
in which it is assumed that all correct nodes initialize the agreement procedure (and any underlying 
primitives) at about the same time. By assuming concurrent initializations of the algorithm a synchronous 
rounds structure can be enforced and used to infer on the progression of the algorithm from the point of 
initialization. Moreover, there is always an implicit assumption about the consistency of the initial states 
of all correct nodes, or at least a quorum of them. 

We consider a more severe fault-model in which in addition to the permanent presence of Byzantine 
failures, the system can also be subject to severe transient failures that can temporarily throw all the 
nodes and the communication subsystem out of the assumption boundaries. E.g. resulting in more than 
one third of the nodes being Byzantine or messages of non-faulty nodes getting lost or altered. This will 
render the whole system practically unworkable. Eventually the system must experiences a tolerable level 
of permanent faults for a sufficiently long period of time. Otherwise it would remain unworkable forever. 
When the system eventually returns to behave according to the presumed assumptions, each node may be 
in an arbitrary state. It makes sense to require a system to resume operation after such a major failure 
without the need for an outside intervention to restart the whole system from scratch or to correct it. 

Classic Byzantine algorithms cannot guarantee to execute from an arbitrary state, because they are 
not designed with self-stabilization in mind. They typically make use of assumptions on the initial state 
of the system such as assuming all clocks are initially synchronized or that the initial states are initialized 
consistently at all correct nodes (cf. from the very first polynomial solution [10] through many others 
like [14]). Conversely, A self-stabilizing protocol converges to its goal from any state once the system 
behaves well again, but is typically not resilient to the permanent presence of faults. 

In trying to combine both fault models, Byzantine failures present a special challenge for designing 
self-stabilizing distributed algorithms due to the "ambition" of malicious nodes to incessantly hamper sta- 
bilization. This difficulty may be indicated by the remarkably few algorithms resilient to both fault models 
(see [4] for a review). The few published self-stabilizing Byzantine algorithms are typically complicated 
and sometimes converge from an arbitrary initial state only after exponential or super exponential time 
([8]). Recently efficient solutions were presented for the strict synchronization model in which an outside 
entity provides repetitive synchronized timing events at all correct nodes at once ([9]). 

In our model correct nodes cannot assume a common reference to time or even to any common anchor 
in time and they cannot assume that any procedure or primitive initialize concurrently. This is the result of 
the possible loss of synchronization following transient faults that might corrupt any agreement or coordi- 
nation among the correct nodes and alter their internal states. Thus synchronization must be restored from 
an arbitrary state while facing on-going Byzantine failures. This is a very tricky task considering that all 
current tools for containing Byzantine failures, such as [2, 14], assume that synchronization already exists 
and are thus preempted for use. Our protocol achieves self-stabilizing Byzantine agreement without the 
assumption of any existing synchrony besides bounded message delivery. In [1] it is proven to be impossible 
to combine self-stabilization with even crash faults without the assumption of bounded message delivery. 

Note that the problem is not relaxed even in the case of a one-shot agreement, i.e. in case that it 
is known that the General will initiate agreement only once throughout the life of the system. Even if 
the General is correct and even if agreement is initiated after the system has returned to its coherent 
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behavior following transient failures, then the correct nodes might hold corrupted variable values that 
might prevent the possibility to reach agreement. The nodes have no knowledge as to when the system 
returns to coherent behavior or when the General will initiate agreement and thus cannot target to reset 
their memory exactly at this critical time period. Recurrent agreement initialization by the General allows 
for recurrent reset of memory with the assumption that eventually all correct nodes reset their memory in 
a coherent state of the system and before the General initializes agreement. This introduces the problem 
of how nodes can know when to reset their memory in case of many ongoing concurrent invocations of 
the algorithm, such as in the case of a faulty General disseminating several values all the time. In such 
a case correct nodes might hold different sets of messages that were sent by other correct nodes as they 
might reset their memory at different times. 

In our protocol, once the system complies with the theoretically required bound of 3/ < n permanent 
Byzantine faulty nodes in a network of n nodes and messages are delivered within bounded time, following 
a period of transient failures, then regardless of the state of the system, the goal of Byzantine agreement is 
satisfied within O(f') communication rounds (where /' < / is the actual number of concurrent faults). The 
protocol can be executed in a one-shot mode by a single General or by recurrent agreement initializations 
and by different Generals. It tolerates transient failures and permanent Byzantine faults and makes no 
assumption on any initial synchronized activity among the nodes (such as having a common reference to 
time or a common event for triggering initialization). 

For ease of following the arguments and proofs, the structure and logic of our SS-Byz- Agree proce- 
dure is modeled on that of [14]. The rounds in that protocol progress following elapsed time. Each round 
spans a constant predefined time interval. Our protocol, besides being self-stabilizing, has the additional 
advantage of having a message-driven rounds structure and not time-driven rounds structure. Thus the 
actual time for terminating the protocol depends on the actual communication network speed and not on 
the worst possible bound on message delivery time. 

It is important to note that we have previously presented a distributed self-stabilizing Byzantine pulse 
synchronization procedure in [3]. It aims at delivering a common anchor in time to all correct nodes 
within a short time following transient failures and with the permanent presence of Byzantine nodes. 
We have also previously presented a protocol for making any Byzantine algorithm be self-stabilizing [5], 
assuming the existence of synchronized pulses. Byzantine agreement can easily be achieved using a pulse 
synchronization procedure: the pulse invocation can serve as the initialization event for round zero of the 
agreement protocol. Thus any existing Byzantine agreement protocol may be used, on top of the pulse 
synchronization procedure, to attain self-stabilizing Byzantine agreement. The current paper achieves 
Byzantine agreement without assuming synchronized pulses. Moreover, we show in [6] that synchronized 
pulses can actually be produced more efficiently atop the protocol in the current paper. This pulse 
synchronization procedure can in turn be used as the pulse synchronization mechanism for making any 
Byzantine algorithm self-stabilize, in a more efficient way and in a more general model than by using the 
pulse synchronization procedure in [3]. 

An early version of the results covered in the current paper appeared in [7]. The current paper provides 
elaborated proofs and correct some mistakes that appear in the early version. 

In [15] it is shown how to initialize Byzantine clock synchronization without assuming a common initial- 
ization phase. It can eventually also execute synchronized Byzantine agreement by using the synchronized 
clocks. The solution is not self-stabilizing as nodes are booted and thus do not initialize with arbitrary 
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values in the memory. 

In [11] consensus is reached assuming eventual synchrony. Following an unstable period with unbounded 
failures and message delays, eventually no node fails and messages are delivered within bounded, say d, 
time. At this point there is no synchrony among the correct nodes and they might hold copies of obsolete 
messages. This is seemingly similar to our model but the solution is not truly self-stabilizing since the 
nodes do not initialize with arbitrary values. Furthermore, the solution only tolerates stopping failures 
and no new nodes fail subsequent to stabilization. Consensus is reached within 0(d). That paper also 
argues that in their model, although with Byzantine failures, consensus cannot be reached within less than 
O(f') • d time, which is essentially identical to our time complexity. Our solution operates in a more severe 
fault model and thus converges in linear time. 

2 Model and Problem Definition 

The environment is a network of n nodes that communicate by exchanging messages. We assume that the 
message passing medium allows for an authenticated identity of the senders. The communication network 
does not guarantee any order on messages among different nodes, though, when the network is functioning 
correctly, any message sent will eventually be delivered. Individual nodes have no access to a central clock 
and there is no external pulse system. The hardware clock rate (referred to as the physical timers) at each 
non-faulty node has a bounded drift, p, from real-time rate. Ensuant to transient failures there can be an 
unbounded number of concurrent faulty nodes, the turnover rate between faulty and non-faulty nodes can 
be arbitrarily large and the communication network may behave arbitrarily. 

Definition 1. A node is non-faulty at times that it complies with the following: 

1. (Bounded Drift) Obeys a global constant < p < 1 (typically p sa 10~ 6 J, such that for every 
real-time interval [u,v] : 

(1 — p)(v — u) < 'physical timer'(v) — 'physical timer\u) < (1 + p)(v — u). 

2. (Obedience) Operates according to the instructed protocol. 

3. (Bounded Processing Time) Processes any message of the instructed protocol within ir real-time 
units of arrival time. 1 

A node is considered faulty if it violates any of the above conditions. A faulty node may recover 
from its Byzantine behavior once it resumes obeying the conditions of a non-faulty node. In order to keep 
the definitions consistent, the "correction" is not immediate but rather takes a certain amount of time 
during which the non-faulty node is still not counted as a correct node, although it supposedly behaves 
"correctly". 2 We later specify the time-length of continuous non-faulty behavior required of a recovering 
node to be considered correct. 

Definition 2. The communication network is non-faulty at periods that it complies with the following: 

1 We assume that the bounds include also the overhead of the operating system in sending and processing of messages. 
2 For example, a node may recover with arbitrary variables, which may violate the validity condition if considered correct 
immediately. 
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1. Any message arrives at its destination node within 5 real-time units; 

2. The sender's identity and the content of any message being received is not tampered. 

Thus, our communication network model is a "bounded-delay" communication network. We do not 
assume the existence of a broadcast medium. We assume that the network cannot store old messages for 
arbitrary long time or lose any more messages, once it becomes non-faulty. 3 

We use the notation d = (5 + ir) x (1 + p). Thus, when the communication network is non-faulty, 
d is the upper bound on the elapsed time from the sending of a message by a non-faulty node until it is 
received and processed by every non-faulty node, as measured by the local clock at any non-faulty node. 4 

Note that n, f and d are fixed constants and thus non-faulty nodes do not initialize with arbitrary 
values of these constants. 

A recovering node should be considered correct only once it has been continuously non-faulty for 
enough time to enable it to have deleted old or spurious messages and to have exchanged information with 
the other nodes. 

Definition 3. The communication network is correct following A net real-time of continuous non-faulty 
behavior. 5 

Definition 4. A node is correct following A n0 ^ e real-time of continuous non-faulty behavior during a 
period that the communication network is correct. 6 

Definition 5. (System Coherence) The system is said to be coherent at times that it complies with the 
following: 

• (Quorum) There are at least n — f correct nodes, 7 where f is the upper bound on the number of 
potentially non-correct nodes at steady state. 

Hence, when the system is not coherent, there can be an unbounded number of concurrent faulty nodes; 
the turnover rate between the faulty and non-faulty nodes can be arbitrarily large and the communication 
network may deliver messages with unbounded delays, if at all. The system is considered coherent, once 
the communication network and a sufficient fraction of the nodes have been non-faulty for a sufficiently 
long time period for the pre-conditions for convergence of the protocol to hold. The assumption in this 
paper, as underlies any other self-stabilizing algorithm, is that the system eventually becomes coherent. 

Definition 6. (System Convergence) The system is said to be stable at times that it complies with the 
following: 

• (converging) The system has been coherent for A st £ time units; 8 

3 A non-faulty network might fail to deliver messages within the bound but will be masked as a fault and accounted for 
in the / faults. Essentially, we assume that messages among correct nodes are delivered within the time bounds. 
"Nodes that were not faulty when the message was sent. 
5 We assume A ne t > d. 
6 A n0( j e is defined in the next section 

7 The condition can be replaced by (n + /)/2 correct nodes with some modifications to the structure of the protocol. 
8 We define in the next section. 
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• (stability) The system remained coherent since that time. 

It is assumed that each node has a local timer that proceeds at the rate of real-time. The actual 
reading of the various timers may be arbitrarily apart, but their relative rate is bounded in our model. To 
distinguish between a real-time value and a node's local-time reading we use t for the former and r for 
the latter. The function rt(r p ) represents the real-time when the timer of a non-faulty node p reads (or 
read) t p at the current execution. 

Observe that the local time at a node may wrap around, since we assume transient faults. The protocol 
and the primitives presented below require measuring only intervals of times. It is assumed that the local 
time wrap around is larger than a constant factor of the maximal interval of time need to be measured. 
This way a node can uniquily measure any necessary intervals of time. 

Since nodes measure only intervals of time that span several d, and d itself includes a worst case drift 
factor, by definition, then d is an upper bound on the time it takes to send and process messages among 
correct nodes, measured by each local timer, i.e., including the drift factor. 

3 The ss-Byz-Agree protocol 

We consider the Byzantine agreement problem in which a General broadcasts a value and the correct nodes 
agree on the value broadcasted. In our model any node can be a General. An instance of the protocol 
is executed per General, and a correct General is expected to send one value at a time. 9 The target is 
for the correct nodes to associate a local-time with the protocol initiation by the General and to agree on 
a specific value associated with that initiation, if they agree that such an initiation actually took place. 
There is a bound on how frequent a correct General may initiate agreements, though Byzantine nodes 
might try to trigger agreements on their values at an arbitrary rate. 

The ss-Byz- Agree protocol is composed of the Agreement procedure (the main body of the protocol) 
and two primitives: the primitive Initiator- Accept and the msgd-broadcast one (as detailed later). 
The General, G, initiates an agreement on a value m by disseminating the message (Initiator, G,m) to 
all nodes. Upon receiving the General's message, each node invokes the ss-Byz-Agree protocol, which 
in turn invokes the primitive Initiator- Accept. Alternatively, if a correct node did not receive the 
General's message but concludes that enough nodes have invoked the protocol (or the primitive) it will 
participate by executing the appropriate parts of the primitive Initiator- Accept (but will not invoke 
it), and following the completion of the primitive that node may participate in the corresponding parts of 
the agreement procedure. 

We will prove the following properties of the ss-Byz-Agree protocol. When the system is stable, 
if all correct nodes invoke the protocol within a "small" time-window, as will happen if the General is a 
correct node, then it is ensured that the correct nodes agree on a value for the General. If the General is 
a correct node, the agreed value will be the value sent by the General. When not all correct nodes happen 
to invoke the ss-Byz- Agree protocol within a small time-window, as can happen if the General is faulty, 
then if any correct node accepts a non-null value, all correct nodes will accept and agree on that value. 

9 One can expand the protocol to a number of concurrent invocations by using an index to differentiate among the 
concurrent invocations. 
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Protocol SS-Byz- Agree on (G,m) /# Executed at node q. r q is the local-time 
at q. */ 

/* Block Q is executed only when (and if) invoked. */ 
/* The rest is executed following a setting of a value to T q . */ 
/* At most one of blocks R through U is executed per such a setting of r 9 G . 

*/ 

QO. If q = G then send (Initiator, G, rn) to all . /* initiation of the 

primitive by the leader */ 

Ql. If received (Initiator, G, m) invoke INITIATOR- Accept(G, m) . 

/# determines T„ and a value m' for node G */ 

Rl. if I-accept (G,m',r^) and r q - t° < 4d then 

R2. value := (G,m'}; 

R3. msgd-broadcast(q, value, 1) ; 

R4. stop and return {value, t^) . 

51 . if by Tq, r q < t° + (2r + 1) ■ $, 

accepted r distinct messages (pi, (G,m") ,i), l<i<r, 
where Vi, j 1 < i,j < r and pi =fc Pj G, then 

52. value := (G,m"}; 

53. MSGD-BROADCAST(q, Value, T + 1) ; 

54. stop and return {value, Tq 3 } . 

Tl. if by T q , T q > r, G + (2r + 1) ■ $, \broadcasters\ < r — 1 then 
T2. stop and return (J-,^* 3 ). 

Ul . if Tq > t° + (2/ + 1) ■ $ then 
U2. stop and return (_L,t,j G ). 

cleanup: 

- Erase any value or message older than (2/ + 1) • $ + 3d time units. 

- 3d after returning a value reset INITIATOR- ACCEPT, r G , and 

MSGD-BROADCAST . 



Figure 1: The SS-Byz-Agree protocol 

For ease of following the arguments and the logic of our ss-Byz- Agree protocol, we chose to follow 
the building-block structure of [14]. The primitive msgd-broadcast, presented in Section 5, replaces 
the broadcast primitive that simulates authentication in [14]. The main differences between the original 
synchronous broadcast primitive and msgd-broadcast are two-folds: first, the latter executes rounds 
that are anchored at some agreed event whose local-time is supplied to the primitive through a parameter; 
second, the conditions to be satisfied at each round at the latter need to be satisfied by some time span 
that is a function of the round number and need not be executed only during the round itself. This allows 
nodes to rush through the protocol in the typical case when messages among correct nodes happen to be 
delivered faster than the worse case round span. 

The SS-Byz-Agree protocol needs to take into consideration that correct nodes may invoke the 
agreement procedure at arbitrary times and with no knowledge as to when other correct nodes may have 
invoked the procedure. A mechanism is thus needed to make all correct nodes attain some common notion 
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as to when the General may have sent a value, and what that value is. The differences of the real-time 
representations of the different nodes' estimations should be bounded. This mechanism is satisfied by the 
primitive Initiator- Accept, presented in Section 4. The use of this initial step in the protocol provides 
the nodes with an initial potential value of the General, and as a result number of "rounds" necessary to 
reach agreement is two less than those of [14]. 

We use the following notations in the description of the agreement procedure and the related primitives: 

• Let <I> be the duration of time equal to {rf kew + 2d) local-time units on a correct node's timer, where 
T skew = 6^ i n the context of this paper. Intuitively, is the duration of a "phase" on a correct 
node's timer. 

• A agr , the upper bound on the time it takes to run the agreement protocol, will be equal to (2/+l)-<J>. 

• Ao = 13d, the minimal time between consecutive invocations of the protocol by the General, for 
different values. 

• A 

rmv — (Aagr + Ao), the time after which old values are decayed. 

• A v = (15d + 2A rmv)i the minimal time between two invocations of the protocol by the General, for 
the same value. 

• A noc j e = A v + A a g r , the time it takes for a non-faulty node to be considered correct. 

• A reset = 20d + 4A rmv , the time during which the General sends nothing, when it notices a failure 
in agreeing on a value it sent. 

• A st b = 2A reset , stabilization time of the system. 

• _L denotes a null value. 

• In the primitive Initiator- Accept: 

- An l-accept 10 is issued on values sent by G. 

- Tg denotes the local-time estimate, at node q, as to when the General has sent the value 
associated with the I-accept by node q. 

In the context of this paper we assume that a correct General conform with the following criteria when 
sending its messages. 

Sending Validity Criteria: A non-faulty General G sends (Initiator, G, m) provided that: 

[I G 1] At least Ao time passed from the sending of the previous initiation message by G. 

[IG2] At least A v time passed from the sending of previous initiation message with the same value m by 
G. 

Notice that both limitations can be circumvented by adding counters to concurrent agreement initia- 
tions. The difference between the two cases has to do with the ability to converge from an arbitrary initial 
state. If a node can send the same message again and again repeatedly, there is a way for the adversary 
to confuse of convergence protocol, as can be seen in the next section. 

10 An accept is issued within msgd-broadcast. 



8 



Definition 7. We say: 



• A node p decides at time r if it stops at that local-time and returns value . 

• A node p aborts if it stops and returns _L . 

• A node p returns a value if it either aborts or decides. 

The ss-Byz- Agree protocol is presented (see Figure 1) in a somewhat different style than the original 
protocol in [14]. Each round has a precondition associated with it: if the local timer value associated with 
the initialization by the General is defined and the precondition holds then the step is to be executed. It 
is assumed that the primitives' instances invoked as a result of the SS-Byz- Agree protocol are implicitly 
associated with the agreement instance that invoked them. A node stops participating in the procedures 
once it returns a value and it stopped participating in the invoked primitives 3d time units after that. 
We use the term participate to refer to a node that executes the protocol's (and primitives') steps. The 
term invoke will refer to a node that also executes the first block of the protocol (Block Q) or primitive 
(Block K), as each correct node would do if the General is a correct one. A node accumulates messages 
associated with the protocol even before it invokes it or participates in it. Such messages are decayed if 
the node doesn't invoke or participate in the protocol, or being processed once it does. 

The ss-Byz-Agree protocol satisfies the following typical properties, provided that the system is 
stable: 

Agreement: If any connect node decides (G,m), all correct nodes decide the same; 

Validity: If the General invokes SS-Byz-Agree then each correct node decides on the value sent by G; 

Termination: The protocol terminates in a finite time. 

Note that in light of our definitions, the Agreement property actually says that if the protocol returns 
a value /_L at any correct node, it returns the same value at all correct nodes. 
The SS-Byz-Agree protocol also satisfies the following timing properties: 

Timeliness: 

1. (agreement) If a correct node q decides on (G,m) at r q then any correct node q' decides on (G,m) 
at some r q i such that, 

(a) \rt(r q ) — rt{r q i)\ < 3d, and if validity holds, then \rt{T q ) — rt{T q ')\ < 2d; 

(b) \rt(r^)-rt(r^)\ < 6d; 

(c) rt{T q ),rt{T^) € [t\ — 2d,t 2 ], where [ti,*2] is the interval within which each correct node, p, 
that obtained the t9 appearing in (b) following the invocation of SS-Byz-Agree (G,m), did 
so; 

(d) rt{rf) < rt{T q ) and rt{r q ) - ri(rf) < A agr . 

2. (validity) If all correct nodes invoked the protocol in an interval [to,to + d], as a result of some value 
m sent by a correct General G that conform with the Sending Validity Criteria, then for every correct 
node q, the decision time r q , satisfies to — d < rt(r q ) < rt(r q ) < t + Ad. 
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3. (termination) The protocol terminates within A agr time units of invocation, and within A agr + Id 
in case it was not invoked explicitly. 

4. (separation) Let p and q be two correct nodes that decided on agreements regarding 67, then 

(a) for m ^ m' , \rt(T°) - rt(r°)\ > id; 

(b) for m = m', either |rt(r°) - rt(r°)| < Qd or |rt(r°) - rt(r°)| > 2A rmv - 3d 

Note that the bounds in the above property is with respect to d, the bound on message transmission 
time among correct nodes and not the worse case deviation represented by 

Observe that since there is no prior notion of the possibility that a value may be sent, it might be that 
some nodes associate a _L with a faulty sending and others may not notice the sending at all. 

The proof that the ss-Byz-Agree protocol meets its properties appears in Section 6.3. 

4 The primitive Initiator- Accept 

In a typical agreement protocol a General that wants to send some value broadcasts it in a specific round 
(say the first round of the protocol). From the assumptions on synchrony all correct nodes can check 
whether a value was indeed sent at the specified round and whether multiple (faulty) values were sent. In 
the transient fault model no such round number can be set beforehand adjoined with the broadcast. Thus 
a faulty General has more power in trying to fool the correct nodes by sending its values at completely 
different times to whichever nodes it chooses. 

The primitive Initiator- Accept aims at making the correct nodes associate a local time with the 
invocation of the protocol (and primitive) by (the possibly faulty) General, and to converge to a single 
candidate value for the agreement to come. Since the full invocation of the protocol by a faulty General 
might be questionable, there may be cases in which some correct nodes will return a _L value and others 
will not identify the invocation as valid. But, as we will prove, if any correct node happens to return a 
value 7^_L within a given timeframe, all correct nodes will return the same value. 

In order to initiate the process of broadcasting its value (one value at a time) the General sends 
(Initiator, G, m) to all nodes, provided some validity criteria are met, as we detail below. As a re- 
sponse to that initiation message, each non-faulty node (including the General) invokes the primitive 
Initiator- Accept. Each node dynamically executes the primitive, whenever relevant messages are be- 
ing received, to obtain an estimate to its (relative) local-time at which the primitive may have been initiated. 
The primitive guarantees that all correct nodes' estimates are within some bounded real-time of each other. 

To ensure convergence we need to add to the two Sending Validity Criteria of Section 3 a third one: 

[IG3] No invocation of Initiator- Accept (G, *) failed in the last A reset time, where an invoca- 
tion is considered failed if any of the following is failed: executing lines L4, M4 or N4 of the 
Initiator- Accept primitive (see Figure 2) is not completed within 2d, 3d or Ad of the invoca- 
tion, respectively. 

The General, before initiating the primitive, removes from its memory all previously received messages 
associated with any previous invocation of the primitive with him as a General. 
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Primitive Initiator- Accept (G,m) /* Executed at node q. r q is the local-time at q. 
*/ 

/* Block K is executed only when (and if) the primitive is explicitly invoked. */ 
/* Lines LI through N3 are repeatedly executed upon receiving messages . */ 

Kl. if i_values[G,m'] =± for every m! 7^ m & last q (G) = _L & 

did not send any (support , G , *) in [r q — d, T q ] & /* allow for recent messages */ 

last q (G,m) = ± at r q — d then 
K2 . i_values[G, m] := r q — d; /* recording time */ 

send (support, G,m) to all; last q (G,m) = r q ; 

LI. if received (support , G , m) from > n — 2f distinct nodes 

in the interval [r q — ct,T q ] for a < 4d then /* shortest interval */ 

L2. i_values[G, m] := max{i_ values[G, m], (r 9 — a — 2d)}; last q (G,m) = T q ; /* recording 
time */ 

L3. if received (support , G , m) from >n — f distinct nodes 

in the interval [r q — 2d, r q \ then 
L4. send (approve, G,m) to all; last q (G,m) = r q ; /* if not recently sent */ 

Ml. if received (approve, G,m) from > n — 2/ distinct nodes 
in the interval [t 9 — 5d, Tq] then 

M2. ready Q m =' true ' ; last q (G,m) = r g ; 

M3. if received (approve, G,m) from >n — / distinct nodes 

in the interval [r q — 3d, r a ] then 
M4. send (ready, G,m) to all; last q (G,m) = r q ; 

Nl . if ready G m & received (ready, G,m) from > n — 2/ distinct nodes then 
W2. send (ready, G,m) to all; last q (G,m) = T q ; 

N3. if ready q m & received (ready, G,m) from ~>n — f distinct nodes then 

N4. Tg := i_ values[G, m] ; i_values[G, *] := _L; 

remove all (G,m) messages and ignore all (G,m) messages for 3d; 
I-accept (G, m, T g ); last q (G,m) = r q ; last q (G) :— r q . 

cleanup: 

Remove any value or message that is older than Armv time units; 

If last q (G) > r q or last q (G) < T q - (A - 6d) then last q (G) :=_L . 

If last q (G,m) > Tq or last q (G,m) < T q — (2 Armv + 9d) then last q (G,m) :=_L . 



Figure 2: The primitive Initiator- Accept 

Each correct node records the local-time at which it receives each message associated with the invo- 
cation of the primitive, for the specific General. Whenever a new message arrives the node records it and 
its time of arrival. The node goes through the primitive and considers all the various lines of the primitive, 
one by one, and acts accordingly. Notice that the node processes all messages, even if it did not invoke 
the primitive. 

We say that a node does an I-accept of a value sent by the General if it accepts this value as the 
General's initial value, and r° is the estimated local-time at q associated with the initiation of the primitive 
by the General. 

Each node maintains a list /_ values[G, *] for the possible concurrent values sent by the General G, 
where each non-empty entry is a local-time associated with the possible invocation of the primitive with 
that entry value. The list should contain at most a single value if the General it correct. Each node also 
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maintains for each non empty entry a time variable, last(G,m), that indicates the latest time at which 
any stage of the primitive was executed regarding the specific value m. To ensure the compliance of the 
General with the rules of initiating the primitive each node also maintains an additional time variable, 
last(G), measuring the minimal time between two consecutive invocations of the primitive by the General. 

Each entry has an expiration time, and messages have a decay time, so after some time all residue 
of previous invocations are removed. The variables are set to 1 as a result of resetting them. The 
Initiator- Accept primitive requires the knowledge of the state of the vector i_values[G,*] d time 
units in the past. It is assumed that the data structure reflects that information. 

Definition 8. The data structure of a node is fresh with respect to a value m if d units of time ago 
i_values[G, *] did not contain any value and the time variables last(G,m), and last(G) both were _L. 

Thus, as we prove later on, when the data structure is fresh and a correct node receives an initiation 
message form a correct G it will be able to execute successfully Block K of the Initiator- Accept 
primitive. 

Before stating the properties that the primitive Initiator- Accept satisfies we give some intuition 
regarding it. The primitive is composed of five sections: four of them are commands to execute in response 
of receiving messages and the final one is a cleanup process that is carried on in the background. 

Block K states the rules for the invocation of the primitive. It is executed as a result of receiving a 
(Initiator, G, m) message from G. 

Line Kl lists the tests a node carries to ensure that G respects the Sending Validity Criteria. The nodes 
tests whether any other broadcasts of messages were processed not too long ago. Since the message from 
G may take d to arrive, and responses to such a message from other correct nodes may have been received 
already. Therefore the node checks what was the status of its data structure d time units ago. It checks 
whether it recently responded to any initiation message or whether it processed the relevant message from 
other nodes only in the last d units of time. 

Line K2: the node sends its support message to all nodes, and marks the time of sending. The sending 
event entry is marked as a time prior to the invocation of the primitive, therefore d is reduced. 

Block L intends to capture the fact that enough correct nodes have sent the support messages within 
a short period of one another. If that happens an approved message is being produced. 

Line LI: The node tests whether at least one correct node has invoked the primitive in the last 4d 
time units. 

Line L2: The node marks that latest such event. The node reduces 2d to mark a time prior to an 
invocation event would G was a correct node. 

Line L3: The node checks whether at least t + 1 correct nodes have sent support within d of each 
other. Notice that since some messages may take time to arrive and some may take d the interval is 2d. 
Notice also that if at some correct node this is true, at all correct nodes the test of Line LI is true. 

Line L4: Since the node knows that every correct node will end up executing Line L2, and d after that 
all will have Line L3 enabled, it is safe to send an approve message. 

The general controls the previous blocks be deciding when to send the invocation messages to which 
correct node. We now moved to two stages that are controlled by the correct nodes that send the approval 
messages, and there is a need to prevent transient messages that may happen to be in the memory of the 
correct nodes from separating the agreement among correct nodes. 
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Block M intends to verify that all correct nodes have moved a stage before we move to the acceptance 
stage. If enough correct have sent approve within a small time window a ready message will be produced. 

Line Ml: The node tests whether at least one correct node has sent a recent approve. 

Line M2: In such a case, the correct node marks that by setting the ready variable, which will mark 
its potential readiness to move to the final stage and to to join others in Line N2. 

Line M3: The node checks whether every correct node will notice the sending of an approve message. 

Line M4: In such a case the node sends a ready message and move to the final stage. 

Block N is the only block that is not timed by a short interval, in order to enable nodes that may be 
initial spread around to collect their actions. If enough have noticed the readiness to accept the message 
by the general, all will. 

Line Nl: The node tests whether at least one correct node has sent a ready message and whether it 
is ready to move to the final stage. 

Line N2: In such a case, the correct node amplifies the sending and sends its own ready message. 

Line M3: The node checks whether every correct node will notice the sending of ready messages. 

Line M4: in such a case the node set the potential time of the invocation of the protocols by G and 
accepts the sending. In order to prevent recurrence accepting the node clear messages and ignore messages 
for a short time period. 

Block Cleanup has three parts. Any message that is too old is removed. The other two parts rest the 
two variables that measure the elapse time between two consecutive invocations of the same value and 
of different values. The reason that the expiration of last q (G,m) is almost twice A rmv is to separate 
consecutive sending of the same value from the possible transient messages at startup. 

Recall that a node is required to keep time stamps associated with the various entries in its data 
structures and the messages it has received. Each time-stamped entry that is clearly wrong, with respect 
to the current clock reading of r q , is removed; i.e., future time stamps or too old time stamps. 

The primitive Initiator- Accept satisfies the following properties, provided that the system is stable: 
[IA-1] (Correctness) If a correct General G invokes Initiator- Accept (G,m) at to then: 

[IA] All correct nodes I-accept (G,m,T G ) within Ad time units of the invocation; 

[IB] All correct nodes I-accept (G,m,T G ) within 2d time units of each other; 

[IC] For every pair of correct nodes q and q' that I-accepts (G, m, r°) and (G, m, rS), respectively: 

\rt(T$)-rt(r G )\<d; 

[ID] For each correct node q that I-accepts (G, m, T q ) at T q , to — d < rt{r q ) < rt(r q ) < to + 4d. 

[IA-2] (Unforgeability) If no correct node invokes Initiator- Accept (G,m), then no correct node 
I-accepts (G, m, r G ). 

[IA-3] (A agr -Relay) If a correct node q I-accepts {G,rn,T q ) at real-time t, such that t — rt{r G ) < A agr , 
then: 
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[3A] Every correct node q' I-accepts (G, m, t3), at some real-time t', with \t — t'\ < 2d and 

\rt(T°)-rt(rf)\<6d; 

[3B] Moreover, for every correct node q' , rt(rS) < ^2, where some correct node invoked the primitive 

Initiator- Accept at t% 

[3C] For every correct node q', rt(r5) < rt{r q i) and rt{r q i) — rt(r5) < A agr + 8d. 

[IA-4] (Uniqueness) If a correct node q I-accepts (G, m, r°), and a correct node I-accepts (G, m',r° ), 
then 

[4A] for m ^ m', |rt(r, 3 G ) - rt(r°)\ > Ad; 

[4b] for m = m', either |ri(T g G ) - rt(r°)\ < 6d or |rt(r°) - rt(r°)\ > 2A rmv - 3d. 

When the primitive is invoked the node executes Block K. A node may receive messages related to the 
primitive, even in case that it did not invoke the primitive. In this case it executes the rest of the blocks 
of the primitive, if the appropriate preconditions hold. A correct node repeatedly executes each line until 
it execute Line N4. So we assume that a node may send the same message several times. We ignore 
possible optimizations that can save such repetitive sending of messages. Once a node executes Line N4 
it removes all associated messages and ignores related messages for some time, so Line-N4 is not executed 
more than once per execution of the primitive. 

Notice that since Block N is not timed, its expiration is determined by the expiration of old messages, 
which leads to the definition of A rmv . and A v . Following the completion of ss-Byz- Agree, the data 
structures of the related Initiator- Accept instance are reset. 

The proof that the Initiator- Accept primitive satisfies the [IA-*] properties, under the assumption 
that n > 3/, appears in Section 6.1. The proofs also show that from any initial state, after A st b the 
system becomes stable. 

5 The msgd-broadcast Primitive 

This section presents the MSGD-BROADCAST (a message driven broadcast) primitive, which accepts 
messages being broadcasted. The primitive is invoked within the SS-Byz-Agree protocol presented in 
Section 3. The primitive follows the broadcast primitive of Toueg, Perry, and Srikanth [14]. In the original 
synchronous model, nodes advance according to rounds that are divided into phases. This intuitive lock- 
step process clarifies the presentation and simplifies the proofs. Here the primitive MSGD-BROADCAST is 
presented without any explicit or implicit reference to absolute time or round number, rather an anchor 
to the potential initialization point of the protocol is passed as a parameter by the calling procedure. The 
properties of the Initiator- Accept primitive guarantee a bound between the real-time of the anchors of 
the correct nodes. Thus a general notion of a common round structure can be implemented by measuring 
the time elapsed since the anchor. 

In the broadcast primitive of [14] messages associated with a certain round must be sent by correct 
nodes at that round and will be received, the latest, at the end of that round by all correct nodes. In 
msgd-broadcast, on the other hand, the rounds progress with the arrival of the anticipated messages. 
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Primitive msgd-broadcast (p, m, k) 

/* Executed per such triplet at node q. */ 

/* Nodes send specific messages only once. */ 

/* Nodes execute the blocks only when r G is defined. */ 

/* Nodes log messages until they are able to process them. */ 

/* Multiple messages sent by an individual node are ignored. */ 

At node q—p: /* if node q is node p that invoked the primitive */ 



V. node p sends (init,p,m, k) to all nodes; 

Wl . At time r q : r q < r q + 2k ■ $ 

W2. if received (init,p,m,k) from p then 

W3. send (echo,p,m,k) to all; 

XI . At time r q : r q < t° + (2k + 1) • $ 

X2 . if received (echo,p,m, k) from >n — 2f distinct nodes then 
X3. send (init' ,p,m, k) to all; 

X4. if received (echo,p,rn,k) messages from > Jl — / distinct nodes then 
X5 . accept (p, m,k); 

Yl . At time r q : r q < t° + (2k + 2) ■ $ 

Y2. if received (init' ,p,m,k) from > n — 2/ then 

Y3. broadcasters := broadcasters U {p}; 

Y4. if received (init' ,p,m,k) from > n — f distinct nodes then 
Y5. send (echo' ,p,m, k) to all; 

Zl . At any time: 

Z2 . if received (echo' ,p,m,k) from >n — 2f distinct nodes then 
Z3. send (echo' ,p,m, k) to all; 

Z4. if received (echo' ,p,m,k) from >n — f distinct nodes then 

Z5 . accept (p, m,k)\ /* accept only once */ 



cleanup: 

Remove any value or message older than (2/ + 3) • $ time units. 

Figure 3: The MSGD-BROADCAST primitive with message-driven round structure 

Thus for example, if a node receives some required messages before the end of the round it may send next 
round's messages. The length of a round only imposes an upper bound on the acceptance criteria. Thus 
the protocol can progress at the speed of message delivery, which may be significantly faster than that of 
the protocol in [14]. 

Note that when a node invokes the primitive it evaluates all the messages in its buffer that are relevant 
to the primitive. The msgd-broadcast primitive is executed in the context of some initiator G that 
invoked ss-Byz- Agree, which makes use of the msgd-broadcast primitive. No correct node will 
execute the msgd-broadcast primitive without first producing the reference (anchor), r G , on its local 
timer to the time estimate at which G supposedly invoked the original agreement. By IA-3A this happens 
within Qd of the other correct nodes. 

The synchronous Reliable Broadcast procedure of [14] assumes a round model in which within each 
phase all message exchange among correct nodes take place. The equivalent notion of a round in our 
context will be $ defined to be: $ := t° + 2d. 
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The MSGD-BROADCAST primitive satisfies the following [TPS-*] properties of Toueg, Perry and 
Srikanth [14], which are phrased in our system model. 

TPS-1 (Correctness) If a correct node p msgd-broadcast (p, m, k) at r p , where r p < r p G + (2k — 1) • $, 
on its timer, then each correct node q accepts (p, m, k) at some r q , r 9 < t 9 g + (2k + 1) - $, on its 
timer and \rt(r p ) — rt(T q )\ < 3d. 

TPS-2 (Unforgeability) If a correct node p does not MSGD-BROADCAST (p,m,k), then no correct node 
accepts (p, m, k). 

TPS-3 (Relay) If a correct node q\ accepts (p,m,k) at t±, T\ < rj 3 + r ■ on its timer then any other 
correct node q2 accepts (p,m, k) at some T2, T2 < t§ + (r + 2) ■ <£, on its timer. 

TPS-4 (Detection of broadcasters) If a correct node accepts (p,m,k) then every correct node q has p € 
broadcasters at some r q , r, < r 9 G + (2k + 2) • <£, on its timer. Furthermore, if a correct node p 
does not msgd-broadcast any message, then a correct node can never have p € broadcasters. 

Note that the bounds in [TPS-l] are with respect to d, the bound on message transmission time among 
correct nodes. 

When the system is stable, the msgd-broadcast primitive satisfies the [TPS-*] properties, under the 
assumption that n > 3f. The proofs that appear in Section 6.2 follow closely the original proofs of [14], 
in order to make it easier for readers that are familiar with the original proofs. 

6 Proofs 

Note that all the definitions, theorems and lemmata in this paper hold only from the moment, and as long 
as, the system is stable. 

6.1 Proof of the Initiator- Accept Properties 

In the proof we distinguish between the initiation of the primitive Initiator- Accept by the General that 
is done by sending (Initiator, G, m) to all nodes, and the invocation of the primitive Initiator- Accept 
by the non-faulty nodes as a result of receiving the above message. Notice that the General himself plays 
a double role; it also invokes the primitive. 

Nodes continuously run the primitive, in the sense that for each incoming message the various "if 
statements" are tested. We say that a node executes a line in the code when the appropriate "if condition" 
holds. In the proofs below, we omit the reference to (G, m) when it is clear from the context. Thus, when 
we refer to a node executing a line it is assumed that it is with (G,m) and that the "if" condition holds. 

Claim 1. If a correct General G doesn't initiate Initiator- Accept in an interval [t — A reset ,t) then, 

1. at t when G initiates the primitive Initiator- Accept with m, all correct nodes will execute 
successfully Line Kl and will send (support, G, m) in the interval [t, i + d]; 

2. byt + Ad all correct nodes will execute Line N4, within 2d of each other; 
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3. at any t' > t, if the correct G initiates its Initiator- Accept with value m' and G did not initiate 
any Initiator- Accept in the interval [t' - A , t') and G did not initiate any Initiator- Accept 
with m' in the interval [t 1 — A v , t') then all correct nodes will execute successfully Line Kl and will 
send (support, G,m') in the interval [t',t f + d], and byt' + 4d all correct nodes will execute Line N4, 
within 2d of each other. 

Proof. Notice that (support, G,m) messages are sent only as a result of receiving the initiation message 
from the General. Recall that A v = 15d+2A rmv and A reset = 20d+4A rmv . Define t = t-20d-4A rmv . 
In the proof we consider only nodes that are correct from time t on. At t + d some correct nodes may 
still end up executing (successfully 11 ) Block K and may end up sending (support, G,m), because of some 
presumably previously received messages; but past t + d, by the code of the primitive, no correct node 
would execute it any more. The last (support, G,m) message resulting from that activity may reach some 
non-faulty node the latest by t + 2d. For that reason, past t + 6d, no correct node will execute Block L 
until a new initiation message will be received by some correct node. 

The latest (approve, G,m) may be sent by t + 4d and reach others by t + 5d. But past t + lOd, 
no correct node will execute Block M. Notice that faulty nodes may still influence some correct nodes to 
execute Block N and it might be that some and not all correct nodes will follow them. 

By t + lOd + A rmv the variable ready,. m (for all possible values of m) will decay at all correct nodes 
and none will execute Block N or update last(G, m) anymore. By t + lOd + 2A rmv + d no correct node 
will hold in its memory any message claimed to be sent by a correct node and all variables in all data 
structures, including last q (G), will decay. The variable last q (G,m) will decay at all correct nodes by 
t + lOd + 2A rmv + 2A rmv + 9d = t + I9d + 4A rmv = t + A rese t - d. 

Therefore, if at time t the correct G will initiate Initiator- Accept with any m, all correct nodes 
will execute successfully Line Kl and will send support within d of each other, completing the proof of 
the first item of the claim. 

To prove the second item of the claim, notice that by t + 2d all correct nodes will execute successfully 
Line L4, and by t + 3d all will execute successfully both lines M2 and M4. By t + \d all will execute 
successfully Line N4. Let q be the first correct node executing Line N4 at some time t\ in this interval, 
following its execution of lines M4 and N3. By t\ + d all will execute Line N2 and by t\ + 2d all will 
execute Line N4, and will set the value of last(G,m) and last q (G). 

To prove the third item of the claim we will use a mathematical induction on the initiations of 
Initiator- Accept past time t. Since the correct G initiates Initiator-Accept sequentially, the 
order of initiations is well defined. Let i, i > 0, be the index describing the order of initiations past time 
t. Case i = holds by the first two items of the claim. 

Assume that the third item holds for i— 1 and prove it for i. Let t be the time at which the i—1 initiation 
started. By the induction hypothesis, and by the code of the primitive, by t + 4d + Ao — Qd < t + Ao all 
will reset last q (G). Therefore, by t' all non-faulty have reset the value of last q (G). If G did not initiate 
Initiator- Accept with m' after time t' — A v , then by the proof of the first item of the claim, all will 
execute Block K. The flow of the proof of the second item of the claim completes the proof. 

Otherwise, let to, t > t, be the last time G initiated Initiator- Accept with m'. By the induction 
hypothesis, by to + 4d all non-faulty nodes will execute Line N4 and by to + Ad + 2A rmv + 9d all would 

n We omit the term "successfully" from now on 
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have reset last(G, ml). Since to + Ad + 2A rmv + 9d < to + A v < t', again, all will execute Block K, and 
following the arguments of the first two items, the claim holds. □ 

The proof can be extended to prove the following corollary for non-faulty nodes that become correct. 

Corollary 1. Claim 1 holds for any set of at least n — f — 1 nodes and a General that are all non-faulty 
from time t — A reset on. 

In the proofs below we need to refer to the coherency of the system and to the minimal time past from 
the time the network becomes correct. We denote by lq the time by which the network becomes correct 
and there are at least n — f non-faulty nodes that remain non-faulty from that time on. The system is 
considered stable from time i\ = in + 2A reset , and as long as the system remains coherent. 

In the rest of this section, in all the claims and proofs, whenever we refer to a non-faulty node we 
imply a non- faulty node that remains non-faulty from time to on. 

Lemma 1. Once the system is stable, at any time past time i\, if a correct General G initiates the primitive 
Initiator- Accept at some time t, not sooner than An of the beginning of the previous initiation, and 
not sooner than A v of the last initiation with the the same value m, then within d of the initiation, all 
correct nodes will send {support, G, m) . Moreover, byt + Ad all correct nodes will execute Line N4, within 
2d of each other. 

Proof. Recall that lq is the time by which the network became correct, as defined above. Before in every 
non-faulty node may have arbitrary values in the various variables of Initiator- Accept and some of 
the messages being accumulated may be a result of the transient fault. 

Past io + d all received messages claimed to be sent by non-faulty nodes were actually sent by non-faulty 
nodes. Observe that messages resulting form the initial arbitrary state may be sent by non-faulty nodes 
as a result of their initial state without actually receiving the required messages, since such messages may 
be in their initial memory state. 

Past Lq + 6d, whenever a non-faulty nodes considers support or approve messages that were re- 
ceived within the appropriate time intervals in Block L and Block M of the primitive it considers only 
messages from non-faulty nodes that were sent by non-faulty nodes as a result of executing the code of 
Initiator- Accept. 

Past to, if a non-faulty General G doesn't initiate Initiator- Accept in an interval [t,t + A reS et). 
where t + A reset < i, by Claim 1 the lemma holds. 

Now assume that the non-faulty node G did initiate Initiator- Accept in the interval [lq, i.n+A reset ). 
If during any such invocation (when executing Initiator- Accept as one of the participating nodes) G 
fails to successfully execute either Line L4 within 2d of the invocation, or Line M4 within 3d of the 
invocation or Line N4 within Ad of of the invocation, then it will not initiate the primitive for another 
Areset. and by Claim 1 the lemma holds. 

The only case that is left is when G did initiate Initiator- Accept in the interval [lq, lq + A reset ) = 
[z-o, to + 20d + 4A rmv ) and whenever it does so, it successfully executes Line L4, Line M4, Line N4 within 
2d, 3d, and Ad, respectively. Recall that before initiating the primitive a non-faulty General removes all 
past messages associated with the primitive. Let i > to + d be a time at which G invoked the primitive. 
Therefore, past time i, all messages from non-faulty nodes that G receives, while executing the primitive, 
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were actually sent by non-faulty nodes. By assumption, by i + 2d G executes Line L4, therefore, by the 
code of the primitive, by i + 3d all non-faulty nodes would have i_values[G,m] defined. Similarly, by 
t + 3d G execute Line M4, therefore by t + Ad all will have ready Gm ='true'. Since all the (ready, G,m) 
messages G accumulates were actually sent past i — d (it may receive these messages after invoking the 
primitive), all non-faulty nodes will receive at least t + 1 of them by t + 5d, and by t + 6d all non-faulty 
nodes will successfully execute Line N4. 

Let t' be the first time, past to + 20<i + 4A rmv , at which G, as a correct node invokes the primitive 
with some m, assuming it didn't do so with that specific m for at least A v = 15d + 2A rmv , and for any 
other m! for at least Ao = 13d. Let i be the last time G invoked the primitive with that specific m. By the 
arguments above, by i+6d all non-faulty nodes would have set last(G,m), and by t + 6d+2A rmv + 9d < t' 
all would have reset it. For similar reasons, if t was the last time prior to t' at which G invoked the primitive 
with any value, then by t + 6d all would have executed Line N4, and by t + 6d + Ao — 6d = t + Ao < t' 
would have reset the variable last q (G). Therefore, when each correct node receives the invocation it will 
send (support, G,m) within d of each other and by t! + Ad all non-faulty nodes will execute Line N4, 
within 2d of each other. 

To complete the proof we use mathematical induction as was done in the proof of Claim 1. □ 

Lemma 1 and the validity criteria of initiating the primitive Initiator- Accept imply the following. 

Corollary 2. Once the system is stable, whenever a correct General G initiates the Initiator- Accept 
with some value m, the data structures at all correct nodes is fresh. 

We now prove some technical claims that cover the case of a faulty General. 

Claim 2. If a non-faulty node executes Line M2 (or Line M4) with some (G,m) at some time t, for 
t > lq + 10d, then no non-faulty node will execute Line M2 (or Line M4) with (G,m) at any t', t' E 
(t + lOd, t + 2A rmv ) and in the interval t' E (t,t + 2A rmv + lOd) there is a sub-interval of length at least 
2A rmv during which no non-faulty node executes Line M2 (or Line M4) with (G,m). 

Proof. A non-faulty node that executed Line M2 (or Line M4) with (G,m) at time t has considered only 
messages sent past to + d and noticed at least one message from a non-faulty node, say q, that has sent 
(approve, G,m) at some time in the interval [t — 6d,t]. The non-faulty node q sent the (approve, G,m) 
message as a result of executing Line L4 at some time t' in the above interval. Since q have received n — f 
(support, G,m) messages in the interval [t' — 2d,t'], every non-faulty node should have noticed at least 
t + 1 of these in some interval [t' — 3d,t' + d] and would have executed Line L2 in that interval. This 
implies that all non-faulty nodes have set last(G, m) at some time in the interval [t — 9d, t + d] . By the 
protocol, no non-faulty node will send any (support, G,m) later than t + 2d (it allows for recent messages 
which causes it to send its support a d later) until it will reset last(G,m), which takes 2A rmv + 9d time. 
The earliest this will happen to any non-faulty node is t — 9d + 2A rmv + 9d = t + 2A rmv . 

Since no non-faulty node will send (support, G,m) later than t + 2d, no non-faulty node will execute 
Line L4 later than t + 2d + 2d = t + Ad, and its message may be received by non-faulty nodes by t + 5d. 
Therefore, Line M2 (or M4) may still be executed as a result of such a message as late as t + lOd. This 
implies that no non-faulty node will execute Line M2 (or M4) in the interval (t + lOd, t + 2 A rmv ] . Note 
that by definition 2A rmv > lOd. 
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Observe that the above arguments imply that if t is the latest time in the interval [t, t + Wd] at which 
a non-faulty node executes Line M2, then no non-faulty node will execute Line M2 or Line M4 earlier than 
t + 2A rmv , since each non-faulty node gas set last(G,m) at i — 9d or later. □ 

Corollary 3. If two non-faulty nodes execute Line M2 with (G,m) at some times ti,t2, respectively, for 
h,h > £n + Wd, then either \t\ — £2) < 9d or \t\ — £2) > 2A rmv . 

Claim 3. If a non-faulty node executes Line M4 with (G,m) at some time t, for t > to + Wd, then 
no non-faulty node will execute Line M4 in the interval [t + 8d,t + 2A rmv + 5d]; and in the interval 
[t, t + 2A rmv + 6d], there is sub-interval of length 2A rmv during which no non-faulty node executes either 
Line M2 or Line M4 with (G, m) . 

Proof. A non-faulty node that executed Line M4 with (G,m) at time t has considered only messages 
sent past in + d and noticed at least t + l message from non-faulty nodes that were sent in the interval 
[t — 4d,t]. Each such message is a result of receiving (support, G,m) messages that may have been sent 
as early as t — 7d. Thus, all these are based on actual messages being sent past to + d. 

Let t be a time at which a non-faulty node execute Line M4 with (G,m). By t + d all non-faulty 
nodes will set last(G,m). Past t + 2d and until its last(G,m) is reset no non-faulty node will send 
(support, G,m). Therefore, no non-faulty node will send (approve, G,m) past t + 2d + 2d, and none will 
execute Line M4 past t + Ad + d + 3d and until its last(G,m) is reset. Since a non-faulty node executed 
Line M4 at time t, the set of messages causing it to execute Line M4 should cause all other non-faulty 
node to execute Line M2 at some time past t — M. Thus, this is the earliest time at which some non-faulty 
node may have set last(G,m) and will not set it later. Therefore, no non-faulty node will execute Line M4 
in the interval [t + 8d, t + 2 A rmv + 5d] . 

Observe that the above arguments imply that if i is the latest time the interval [t — 4d,t + 8d] at 
which a non-faulty node executes Line M4, then no non-faulty node will execute Line M2 or Line M4 with 
(G,m) earlier than f + 2A rmv + 5d. □ 

Claim 3 implies the following. 

Corollary 4. If two non-faulty nodes execute Line M4 with (G,m) at some time t\,t2, respectively, for 

t\,t2 > to + Wd, then either \t\ — £2) < 7d or \t\ — £2! > 2A rmv . 

Claim 4. If no non-faulty node executes Line M2 (or Line M4) with (G, m) in an interval (t, t + 2A rmv ], 
for t > in + A rmv , then no non-faulty node will execute Line N2 or Line N4 with (G, m) in the interval 
[t + A rmv , t"\, where t + 2A rmv < t" and some non-faulty node executes Line M4 with (G, m) at time t" . 

Proof. Because t > lq + A rmv , all non-faulty nodes have decayed all messages that appeared as part of 
the initial state that may have not been actually sent. Since we assume that no non-faulty node executes 
Line M2 with (G,m) in the interval (t,t + 2A rmv ], by t + A rmv all will have reset ready Gm and will 
not execute Line N2 or Line N4 any more, so no non-faulty node will send a new ready message. By 
t + 2A rmv , all will decay all previous (ready, G,m) messages that were sent by non-faulty nodes. From 
that time on, even if some non-faulty nodes will execute Line M2, none will be able to execute Line N2 
until a new (ready, G,m) message is produced by a non-faulty node, thus until some non-faulty node 
execute Line M4 with (G,m). □ 
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The proof makes use of the following simple observation. 

Claim 5. At any time t, t > i + A rmv , if a non-faulty node sets i_values[G,m], then some non-faulty 
node has sent (support, G, m) later than rt(i_ values[G, to]). 

Proof. If the node didi it in Line K2, then it trivially holds. Otherwise, the time window considered in 
Line L2 includes a sending event of a correct node, and that happened at the earliest d time units before 
the time window span. □ 

Using the above claims we can now prove the following. 

Lemma 2. Once the system is stable, if any correct node, say q, executes Line N4 with (G, m), at some 
time t, where i — rt(r°) < A rmv — 9d, then 

1. all correct nodes will execute Line N4 with (G, m) within 2d of each other in the interval [i—2d, i+2d]; 

2. for any correct node p, |r£(r°) — rt(r°)| < 6d; 

3. some correct node executed Line M4 later than i — A rmv + 7d 

Proof. Let q be such a correct node. By the condition in Line N3, ready Gm was last set by q while 
executing Line M2 at some time t' , later than i— A rmv . Consider the interval (to + A rmv , i — A rmv — 9d). 
By the definition of stability it is longer than 4A rmv . If no correct node executed Line M4 (with G,m) in 
this interval, since the system is stable, then the preconditions of Claim 4 hold. 

Otherwise, let t\ be the latest time in the above interval at which a correct node executed Line M4. 
By definition \t' — t\\> 9d. Therefore, by Corollary 4 and Corollary 3, \t' — ti\> 2A rmv and this holds for 
any other correct node that executed Line M4 or Line M2 within 9d of q, i.e., within 9d of t'. Therefore, 
again, the preconditions of Claim 4 hold. 

By Claim 5, some correct node have sent (support, G,m) in the interval [rt(T^),t\. By the code of 
the primitive, it would have not done so if any correct node would have executed Line M2 or Line M4 in 
the interval [i — A rmv , rt(r. G ) — 2d], since it would have set its last(G, m) at least d prior to that sending. 

This implies that t' > ri(r°) — 2d, and that any correct node executing Line M2 or Line M4 within 
9d of t' should do so later than t%, where t<i = rt(rP) — 2d > i — A rmv + 7d. 

By Claim 4, some correct node executed Line M4, in the interval [t — A rmv — 9d,t\. Since it should be 
within 9d oft', by the above argument, that should happen at some time £3 in the interval [t2,t\- Proving 
the third item of the claim. By the code of the primitive, every correct node should execute Line M2 in 
the interval [£3 — 5d, £3 + d\. This implies that they should do so in the interval [t2,i+d], which implies 
within the interval I = [t — A rmv + 7d,i + d]. 

The correct node q executed Line N4 at time i. It has received at least t + l (ready, G,m) messages 
from correct nodes. Any correct node sending such a message should have executed Line M2 prior to 
sending the message; and such a message is a result of executing either Line M4 or Line N2. By Claim 4 
that can happen either before time t\ + A rmv or later than time ti. If it would be earlier than t\ + A rmv , 
node q would have decayed that message from its memory since we already argued that \t' — 1\\ > 2A rmv . 

We conclude that all such messages from correct nodes were sent past time £2. Therefore, by t + d 
each correct node would execute Line N2, since its pre-conditions holds, and by i + 2d all will execute 
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Line N4. Let q' be the first correct node to execute Line N4 past time ti- The above arguments imply 
that it has done so in the interval [i — 2d, t + 2d], and that all correct nodes would have executed Line N4 
within 2d of q' . Proving the first item of the claim. 

From the above discussion, some correct node q" executed Line M4 in the interval [t — A rmv + 6d,t\. 
Denote that time by t" . Node q" collected n — / approve messages in the interval [t" — 3d,t"]. At 
least one of which is from a correct node. Let q 1 be that node and let tf be the time it sent its approve 
message. From the above discussion, t' € [i — A rmv + 6d — 3d — d, t\ = [t — A rmv + 2d,t\. Node q' 
collected n — f support messages, with at least n — 2f from correct nodes. Let t\ be the time at which 
the (n — 2/) tft support message sent by a correct node was received by q' . Since q' executed Line L4, all 
these messages should have been received in the interval \t\ — 2d,t\\. Node q' should have set a recording 
time r, rt(r) >t\ — 4d, as a result of (maybe repeating) the execution of Line L2. 

Every other correct node should have received the (n — 2/) th support message sent by a correct node 
at some time in the interval \t\ — d,t\ + d] with the set of in — 2f) support messages sent by correct 
nodes being received in the interval \t\ — 3d, t\ + d]. Each such correct node should have set the recording 
time after (maybe repeatedly) executing Line L2, since this window satisfies the precondition of Line LI. 
Thus, eventually all recording times are >t\ — 5d. Observe that since this interval is short, none of these 
messages would have been decayed by the time they are processed by the correct nodes. 

Some correct node may send a support message, by executing Line K2, at most d time units after 
receiving these n — 2f messages. This can not take place later than t\ + 2d, resulting in a recording time 
of t\ + d, though earlier than its time of sending the support message. This support message (with the 
possible help of faulty nodes) can cause some correct node to execute Line L2 at some later time. The 
window within which the support messages at that node are collected should include the real-time t\ + 3d, 
the latest time any support from any correct node could have been received. Any such execution will 
result in a recording time that is < t\ + 3d — 2d = t\ + d. Thus the range of recording times for all correct 
nodes (including q) are [t\ — 5d, t\ + d\. 

To complete the proof of the second item we need to show that each correct node, p, actually sets 
its Tp. By assumption, i — rt(r°) < A rmv — 9d, therefore rt(r°) > i — A rmv + 9d, This implies that 
rmv + 9d. Implying that t\ — 5d > t — A rmv + 3d. Therefore, when each correct node executes 
Line N4, its r G is well defined, since the i_values[G,m] entry wasn't decayed yet. Thus, completing the 
proof. □ 

We are now ready to prove the properties of the primitive Initiator- Accept. 

Theorem 1. Once the system is stable, the primitive Initiator- Accept presented in Figure 2 satisfies 
properties [IA-1] through [IA-4]. 

Proof. 

Correctness: Corollary 2 proves that when a correct General initiates the primitive, the data-structures at 
correct nodes are fresh. Assume that within d of each other all correct nodes invoke Initiator- Accept 
(G, m). Let t\ be the real-time at which the General invokes its copy of the Initiator- Accept then 
by t2, ti <t± + d, the last correct node did so. Since all data structures are fresh, then no value {G, m'} 
appeared in i_values[G, *] d time units before that, thus Line Kl will hold for all correct nodes. Therefore, 
every correct node sends (support, G,m). Each such message reaches all other correct nodes within d. 
Thus, between t\ and ti + d every correct node receives (support, G,m) from n — f distinct nodes and 
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sends (approve, G,m). By t 2 + 2d every correct node sends (ready, G,m), and by £2 + 3d I-accepts 
(G,m,T'), for some r', thus, proving [IA-1A]. 

To prove [IA-1B], let q be the first to I-accept after executing Line M4. Within d all correct nodes will 
execute Line M2, and within 2d all will I-accept. 

Note that for every pair of correct nodes q and q', the associated initial recording times r and r' satisfy 
\t — t'\ < d. Line K2 implies that the recording times of correct nodes can not be earlier than t\ — d. 
Some correct node may see n — 2f, with the help of faulty nodes as late as t 2 + 2d. All such windows 
should contain a support from a correct node, so should include real-time £2 + d, resulting in a recording 
time of t 2 — d. Recall that t 2 < h + d, proving [IA-1C]. 

To prove [IA-1D] notice that the fastest node may set r' to be t\ — d, but may I-accept only by 
t 2 + 3d < ti + Ad. 

Unforgeability: 

If no correct node invokes Initiator- Accept and will not send (support, G,m), then no correct node 
will ever execute L4 and will not send (ready, G, m). Thus, no correct node can accumulate n — f distinct 
(ready, G,m) messages and therefore will not I-accept (G,m). Moreover, no correct will execute lines 
K2 or L2, and therefore if G is correct, no correct node will invoke Initiator- Accept, and no correct 
will have any entry in the Initiator's data structure. 

A agr -Relay: 

Let q be a correct node that I-accepts (G, m, T q ) at real-time t, such that < t — rt(r q ) < A agr . It 
did so as a result of executing Line N4. By assumption the preconditions of Lemma 2 hold, and therefore 
all correct nodes will I-accept (G,m,Tq) within 2d of each other, in the interval [t — 2d,t + 2d], with r G 
values that are 6d apart. Thus, proving [IA-3A]. 

To prove [IA-3B] notice that any range of messages considered in Line L2 includes a support of a correct 
node. The resulting recording time will never be later than the sending time of the support message by 
that correct node, and thus by some correct node. 

The first part of [IA-3C] is immediate from Line L2 and Line K2. For the second part observe that for 
every other correct node q', rt(r q i) < rt(r q ) + 2d and rt(r°,) > ri(r°) — 6d. Thus, rt(r q i) — rt(r^) < 
rt( Tq ) - rt(r°) + 8d < A agr + 8d. 

Uniqueness: 

To prove [IA-4] observe that the conditions in Line Kl implies that each non-faulty node sends a support 
for a single m at a time. In order to I-accept, a correct node needs to send approve after receiving n — f 
support messages. That can happen for at most a single value of m, because n > 3f. 

By Lemma 2, once a correct node execute Line N4, all do it within 2d. By the protocol, once a node 
decides it removes accepted messages and ignores new message associated with (G,m) for 3d. Therefore, 
all correct nodes issue I-accept, and stop sending messages associated with (G,m) before a correct one 
agrees to consider such messages. So past messages cannot be used again to reproduce another wave of 
decisions, unless a new correct node sends a new support for (G,m). 

Previously sent messages for another value of m will not produce a wave of decisions unless a new 
correct node will send (support, G,m) for such a value. None will send support for a new value for 
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Ao — 6d > 6d, so by the time such a message will be sent, old values will be out of any window of 
consideration for executing any L or M lines of the code by a any correct node. Line N cannot be executed 
unless some correct node excuses Line M4. 

What is left to prove, is that future invocations of the primitive will not violate [IA-4]. 

Again, by Lemma 2, once a correct node execute Line N4, all do it within 2d. Let q be the first to 
excute Line N4 in the current execution, and let it be at time rt(r p ) = t'. By t' + d all non-faulty would 
execute Line L2, and the latest any correct will execute Line K2 is t' + d. By inspecting the possible 
scenarios one can see that no non-faulty will execute Line L2 later than t' + 5d, and the latest value 
set by any correct node in that interval will never be later than t' + d. Thus, for every correct node q, 
ri(r G ) < rt(r p ) + d. 

The earliest time at which any correct node will send (support, G,m) later than that time will be at 
rt(r p ) + Ao — 6d. By inspecting the protocol, the earliest possible setting of value in Line K2 will be to 
rt(r p ) + A — 6c? — 2d. Therefore, if we denote by r timings in the former invocation and by r timings in 
the later one, we conclude that for any two correct nodes p and q, rt(r G 9 ) — ri(r G ) > Ao — 9d = Ad. □ 

We can now state the concluding corollary. 

Corollary 5. The system converge from any initial state within 2 x A reset = d, provided that there are 
n — t non-faulty nodes that are continuously non-faulty during that period. 

Proof. Since all properties hold once the system is stable, and stability is defined as 2 x A reset form the 
time the network is correct, we conclude the proof. □ 

One can reduce the requirement of having the same non-faulty nodes stay continuously so, but we do 
not see this optimization as an important issue. Moreover, the proofs above shows that once a non-faulty 
node discards old values it can be considered correct. Therefore we can state the following corollary. 

Corollary 6. Once the system is stable, a non-faulty node that is non-faulty for A noc y e time, can be 
considered correct. 

6.2 Proof of the msgd-broadcast Properties 

The proofs essentially follow the arguments in the original paper [14]. 

Lemma 3. If a correct node p. L sends a message at local-time Ti, ti < r G + r ■ $ on pi 's timer it will be 
received and processed by each correct node pj at some local-time tj, tj < r G + (r + 1) • 3>, on pj 's timer. 

Proof. Assume that node pi sends a message at real-time t with local-time Ti < r f + r ■ <£. Thus, 
Ti < r G + r(tf KEW + 2d). It should arrive at any correct node pj within d. By IA-3A, r G will be defined 
and the message will be processed no later than by another d. By IA-3A, |rt(r G ) — rt(r G )| < i G EW . Thus, 
rtfr- 3 ) < Tt(r G ) + tf KEV/ , and at time rt(rj), by which the message arrived and processed at pj, we get 

rt( Tj ) < rt(n) + 2d< rt(r G ) + r(t G EW + 2d) + 2d , 

and therefore 

rt{r 3 ) < rt(rf) + t G EW + r(i G EW + 2d) + 2d< rt{rf) + (r + 1) • $ . 

□ 
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Lemma 4. If a correct node ever sends (echo',p, m, k) then at least one correct node, say q', must have 
sent (echo',p, m, k) at some local-time r qh r ql <t° + (2k + 2) • <J>. 

Proof. Let t be the earliest real-time by which any correct node q sends the message (echo' ,p,m, k). If 
t > rt(r q ) + (2k + 2) node q should have received (echo 1 ,p, m, k) from n — 2f distinct nodes, at least 
one of which from a correct node, say q', that was sent prior to local local-time T ql + (2k + 2) • □ 

Lemma 5. If a correct node ever sends (echo' ,p,m, k) then p's message (init,p,m, k) must have been 
received by at least one correct node, say q' , at some time T qh T q , < t2 + 2k ■ <£. 

Proof. By Lemma 4, if a correct node ever sends (echo' ,p, m, k), then some correct node q should send it 
at local-time T q , T q < T G + (2fc+2)-<£. By the primitive MSGD-BROADCAST, q have received (init',p, m, k) 
from at least n — f nodes by some local-time r q , r g < r G + (2k + 2) • At least one of them is a correct 
node q" who have received n — 2f (echo,p, m, k) at some local-time T q// , r q n < r q „ + (2k + 1) • <J>. One of 
which was sent by a correct node q that should have received (init,p, m, k) before sending (echo,p, m, k) 
at some local-time T q , r q < + 2k ■ $. □ 

Lemma 6. If a correct node p invokes the primitive msgd-broadcast (p,m,k) at real-time t p , then 
each correct node q accepts (p, m, k) at some real-time t q , such that \t p — t q )\ < 3d. 

Proof. The init message of p sent in Line V will arrive to every node by t p + d. By IA-3A, by t p + 2d all 
will have their r G defined and will process the init message. By Lemma 3, all will execute Line W3 by 
that time. By t p + 3d all will execute Line X5 and will accept. □ 

Theorem 2. The msgd-broadcast primitive presented in Figure 3 satisfies properties [TSP-1] through 
[TSP-4]. 

Proof. Correctness: Assume that a correct node p MSGD-BROADCASTs (p, m, k) at t p , t p <r p + (2k — 
1) ■ <J>, on its timer. Any correct node, say q, receives (init,p,m, k) and sends (echo,p,m, k) at some T q , 
T q < r q + 2k ■ $ on its timer. Thus, any correct node, say q receives n — f (echo,p,m, k) from distinct 
nodes at some T q , T q < + (2k + 1) • 3>, on its timer and accepts (p,m,k). The second part of the 
correctness is a result of Lemma 6. 

Unforgeability: If a correct node p does not broadcast (p,m,k), it does not send (init,p,m, k), and 
no correct node will send (echo,p, m, k) at some t, r < r G + 2k • <E>, on its timer. Thus, no correct node 
accepts (p,m,k) by r G + (2k + 1) • $ on its timer. If a correct node would have accepted (p,m,k) at a 
later time it can be only as a result of receiving n — f (echo' ,p,m, k) distinct messages, some of which 
must be from correct nodes. By Lemma 5, p should have sent (init,p,m, k), a contradiction. 

Relay: The delicate point is when a correct node issues an accept as a result of getting echo messages. 
So assume that q\ accepts (p,m,k) at t\ = rt(ri) as a result of executing Line X5. By that time it must 
have received (echo,p,m, k) fromn — / nodes, at least n — 2f of them sent by correct nodes. Since every 
correct node among these has sent its message by r G + 2fc-$ on its timer, by Lemma 3, all those messages 
should have arrived to every correct node qi by < r f + (2k + 1) ■ $ on its timer. Thus, every correct 
node qi should have sent (init',p, m, k) at some Tj, n < r G + (2k + 1) • <l>, on its timer. As a result, every 
correct node will receive n — f such messages by some f, f < r G + (2k + 2) • $ on its timer and will send 
(echo',p,m,k) at that time, which will lead each correct node to accept (p,m,k) at a local-time T{. 
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Now observe that all n — 2f (echo,p, m, k) were sent before time t\. By t\ +d they arrive to all correct 
nodes. By t\ + 2d all will have their r G defined and will process them. By t\ + 3d their (init' ,p,m,k) 
will arrive to all correct nodes, which will lead all correct nodes to send (echo' ,p,m, k). Thus, all correct 
nodes will accept (p, m, k) at time n < t\ + Ad. 

By assumption, t\ = rt(n) < rt(rf) + r ■ $. By IA-3A, rt(rf) < rt(rf) + t G EW . Therefore we 
conclude: rtfo) < rt(n) + 4d < rt(rf ) + r • $ + 4d < rt(rf) + i G EW + r ■ $ + 4d < rt(rf) + (r + 2) • 

The case that the accept is a result of executing Line Z5 is a special case of the above arguments. 

Detection of broadcasters: As in the original proof ([14]), we first argue the second part. Assume 
that a correct node q adds node p to broadcasters. It should have received n—2f (init',p, m, k) messages. 
Thus, at least one correct node has sent (init' ,p,m, k) as a result of receiving n — 2/ (echo,p,m, k) 
messages. One of these should be from a correct node that has received the original broadcast message 
of p. 

To prove the first part, we consider two similar cases to support the Relay property. If r = k and the 
correct node, say q, accepts (p,m,k) as a result of receiving (echo,p,m, k) from n — f nodes by some 
T q , r q < r° + (2k + 1) • <J>, on its timer. At least n — 2f of them were sent by correct nodes. Since 
each correct node among these has sent its message at some r, r < r G + 2k • <!>, by Lemma 3, all those 
messages should have arrived to any correct node, say by some Tj, n < rf + (2k + 1) • <J> on its timer. 
Thus, each correct node, say qj should have sent (init' ,p,m, k) at some Tj, Tj < rj 3 + (2k + 1) • <J>, on 
its timer. As a result, by Lemma 3, each correct node, say q' , will receive n — f such messages by some 
Tq/, Tq, < t^ + (2k + 2) ■ $ on its timer and will add p to broadcasters. 

Otherwise, q accepts (p,m,k) as a result of receiving from n — f nodes (echo' ,p,m,k) by some T q 
on its timer. By Lemma 4 a correct node, say qi, sent (echo' ,p, m, k) at some Tj, Tj < rf + (2k + 2) ■ <5. 
It should have received n — f (init' ,p,m,k) messages by that time. All such messages that were sent by 
correct nodes were sent at some r, r < r G + (2k + 1) • <£, on their timers and should arrive at each node 
qj, at some Tj, Tj < rj 3 + (2k + 2) ■ 3>, on its timer. Since there are at least n — 2f such messages, all 
will add p to broadcasters at some r, r < r G + (2k + 2) ■ 3>, on their timers. 

□ 

6.3 Proof of the ss-Byz-Agree Properties 

Theorem 3. (Convergence) Once the system is stable, any invocation of SS-Byz- Agree presented in 
Figure 1 satisfies the Termination property. When n > 3f, it also satisfies the Agreement and Validity 
properties. 

Proof. Notice that the General G itself is one of the nodes, so if it is faulty then there are only / — 1 
potentially faulty nodes. We do not use that fact in the proof since the version of SS-Byz-Agree 
presented does not refer explicitly to the General. One can adapt the proof and reduce A agr by 2 • <I> when 
specifically handling that case. 

By Corollary 2, by the time the system becomes stable, all data structures are fresh. 

We begin by proving Validity. 
Validity: Since all the correct nodes invoke the primitive SS-Byz-Agree as a result of a value sent by 
a correct G, they will all invoke Initiator- Accept within d of each other with fresh data structure, 
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hence [IA-1] implies that they all will execute Block R within 2d of each other, and Validity holds. 

The rest of the proof makes use of the following two lemmata. 

Lemma 7. If a correct node p aborts at local-time r p , r p > r° + (2r + 1) ■ <&, on its timer, then no correct 
node q decides at a time r q , r 9 > t° + (2r + 1) • <J>, on its timer. 

Proof. Let p be a correct node that aborts at time r p , t p > r p G + (2r + 1) • <&. In this case it should 
have identified at most r — 2 broadcasters by that time. By the detection of the broadcasters property 
[TPS-4], no correct node will ever accept (G,m'} and r — 1 distinct messages (qi,m',i) for 1 < i < r — 1, 
since that would have caused each correct node, including p, to hold r — 1 broadcasters by some time 
r, r < t g + (2(r — 1) + 2) • $ on its timer. Thus, no correct node, say q, can decide at a time 
T q > r G + (2r + 1) ■ $ on its timer. □ 

Lemma 8. If a correct node p decides at time t p , r p < r p G + (2r + 1) • <3?, on its timer, then each correct 
node, say q, decides by some time r q , r q < r° + (2r + 3) • <3? on its timer. 

Proof. Let p be a correct node that decides at local-time t p , t p < r p G + (2r + 1) • <&. We consider the 
following cases: 

1. r = : No correct node can abort by a time r, r < r G + (2r + 1) • <£>, since the inequality will not 
hold. Assume that node p have accepted (G,m') by r p < r G + Ad < t g + <£. By the relay property 
[TPS-3] each correct node will accept (G,rn!) by some timer, r < r G + 3-<£ on its timer. Moreover, 
p invokes MSGD-BROADCAST(p, ml, 1), by the Correctness property [TPS-1] it will be accepted by 
each correct node by time r, r < r G + 3 • <J>, on its timer. Thus, all correct nodes will have value ^_L 
and will broadcast and stop by time r G + 3 • <l> on their timers, when executing Block S. 

2. 1 < r < / — 1 : Nodep must have accepted {G,m') and also accepted r distinct (qi,m',i) messages 
for all i,2 < i < r, by time r, r < r G + (2r + 1) • <3?, on its timer. By Lemma 7, no correct node 
aborts by that time. By Relay property [TPS-3] each (qi,m',i) message will be accepted by each 
correct node by some time r, r < r G + (2r + 3) ■ <&, on its timer. Node p broadcasts (p, ml ', r + 1) 
before stopping. By the Correctness property, [TPS-1], this message will be accepted by every correct 
node at some time r, r < t g + (2r + 3) • <J>, on its timer. Thus, no correct node will abort by time 
r, r < t g + (2r + 3) • 3>, and all correct nodes will have value and will thus decide by that 
time. 

3. r = f : Node p must have accepted a (qi,m',i) message for all i, 1 < i < f — 1, by t p , t p < 
t° + (2/ + 1) • $, on its timer, where the / qi's are distinct. If the General G is correct, then by 
Validity the claim holds. Otherwise, at least one of these / nodes (which all differ from G), say 
qj, must be correct. By the Unforgeability property [TPS-2], node qj invoked MSGD-BROADCAST 
{qj,m',j) by some local-time r, r < T G + (2j + l) •<£ and decided. Since j < f the above arguments 
imply that by some local-time r, r < r G + (2/ + !)•$, each correct node will decide. n 
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Lemma 8 implies that if a correct node decides at time r, r < r G + (2r + 1) • $, on its timer, then no 
correct node p aborts at time r p , r p > r p G + (2r + 1) ■ Lemma 7 implies the other direction. 

Termination: Each correct node either terminates the protocol by returning a value, or by time (2/ + 1) • 
+ 3d on its clock all entries will be reset, which is a termination of the protocol. 

Agreement: If no correct node decides, then all correct nodes that execute the protocol abort, and return 
a -1 value. Otherwise, let q be the first correct node to decide. Therefore, no correct node aborts. The 
value returned by q is the value m' of the accepted (p,m',l) message. By [IA-4] if any correct node 
I-accept s, all correct nodes I-accept with a single value. Thus all correct nodes return the same value. 

Timeliness: 

1. (agreement) For every two correct nodes q and q' that decide on (G,m) at r q and r q i, respectively: 

(a) If validity hold, then \rt(r q ) - rt(r q >)\ < 2d, by [IA-3A]; Otherwise, \rt(r q ) - rt(r q >)\ < 3d, by 
[TPS-1]. 

(b) \rt(if) - rt(if)\ < 6d by [IA-3A]. 

(c) rt{rf),rt{T^) € [h - 2d,t 2 ] by [IA-3B]. 

(d) rt(r T G ) < rt(r r ), by [IA-3C], and if the inequality rt(r r ) - rt(r r G ) < A agr would not hold, the 
node would abort right away. 

2. (validity) If all correct nodes invoked the protocol in an interval [to, to+d], as a result of (Initiator, G, m) 
sent by a correct G that spaced the sending by Gd from its last agreement, then for every correct 
node q that may have decided 3d later than G, the new invocation will still happen with fresh 
data structures, since they are reset 3d after decision. By that time it already reset the data 
structures (including latest_accept) of the last execution, and the new decision time r q , satisfies 

as implied by [IA-1D]. 

3. (separation) By [IA-4] the real-times of the I-accepts satisfy the requirements. Since a node will not 
reset its data structures before terminating the protocol, it will not send a support before completing 
the previous protocol execution. Therefore, the protocol itself can only increase the time difference 
between agreements. Thus, the minimal difference is achieved when a decision takes place right 
after the termination of the primitive Initiator- Accept. 

□ 
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